Crate buf_redux

Source
Expand description

Drop-in replacements for buffered I/O types in std::io.

These replacements retain the method names/signatures and implemented traits of their stdlib counterparts, making replacement as simple as swapping the import of the type:

§BufReader:
- use std::io::BufReader;
+ use buf_redux::BufReader;
§BufWriter:
- use std::io::BufWriter;
+ use buf_redux::BufWriter;
§LineWriter:
- use std::io::LineWriter;
+ use buf_redux::LineWriter;

§More Direct Control

All replacement types provide methods to:

  • Increase the capacity of the buffer
  • Get the number of available bytes as well as the total capacity of the buffer
  • Consume the wrapper without losing data

BufReader provides methods to:

  • Access the buffer through an &-reference without performing I/O
  • Force unconditional reads into the buffer
  • Get a Read adapter which empties the buffer and then pulls from the inner reader directly
  • Shuffle bytes down to the beginning of the buffer to make room for more reading
  • Get inner reader and trimmed buffer with the remaining data

BufWriter and LineWriter provides methods to:

  • Flush the buffer and unwrap the inner writer unconditionally.
  • Get the inner writer and trimmed buffer with the unflushed data.

§More Sensible and Customizable Buffering Behavior

Tune the behavior of the buffer to your specific use-case using the types in the policy module:

  • Refine BufReader’s behavior by implementing the ReaderPolicy trait or use an existing implementation like MinBuffered to ensure the buffer always contains a minimum number of bytes (until the underlying reader is empty).

  • Refine BufWriter’s behavior by implementing the [WriterPolicy trait] or use an existing implementation like FlushOn to flush when a particular byte appears in the buffer (used to implement LineWriter).

§Making Room

The buffered types of this crate and their std::io counterparts, by default, use Box<[u8]> as their buffer types (Buffer is included as well since it is used internally by the other types in this crate).

When one of these types inserts bytes into its buffer, via BufRead::fill_buf() (implicitly called by Read::read()) in BufReader’s case or Write::write() in BufWriter’s case, the entire buffer is provided to be read/written into and the number of bytes written is saved. The read/written data then resides in the [0 .. bytes_inserted] slice of the buffer.

When bytes are consumed from the buffer, via BufRead::consume() or Write::flush(), the number of bytes consumed is added to the start of the slice such that the remaining data resides in the [bytes_consumed .. bytes_inserted] slice of the buffer.

The std::io buffered types, and their counterparts in this crate with their default policies, don’t have to deal with partially filled buffers as BufReader only reads when empty and BufWriter only flushes when full.

However, because the replacements in this crate are capable of reading on-demand and flushing less than a full buffer, they can run out of room in their buffers to read/write data into even though there is technically free space, because this free space is at the head of the buffer where reading into it would cause the data in the buffer to become non-contiguous.

This isn’t technically a problem as the buffer could operate like VecDeque in std and return both slices at once, but this would not fit all use-cases: the Read::fill_buf() interface only allows one slice to be returned at a time so the older data would need to be completely consumed before the newer data can be returned; BufWriter could support it as the Write interface doesn’t make an opinion on how the buffer works, but because the data would be non-contiguous it would require two flushes to get it all, which could degrade performance.

The obvious solution, then, is to move the existing data down to the beginning of the buffer when there is no more room at the end so that more reads/writes into the buffer can be issued. This works, and may suit some use-cases where the amount of data left is small and thus copying it would be inexpensive, but it is non-optimal. However, this option is provided as the .make_room() methods, and is utilized by policy::MinBuffered and policy::FlushExact.

§Ringbuffers / slice-deque Feature

Instead of moving data, however, it is also possible to use virtual-memory tricks to allocate a ringbuffer that loops around on itself in memory and thus is always contiguous, as described in the Wikipedia article on Ringbuffers.

This is the exact trick used by the slice-deque crate, which is now provided as an optional feature slice-deque exposed via the new_ringbuf() and with_capacity_ringbuf() constructors added to the buffered types here. When a buffered type is constructed using one of these functions, .make_room() is turned into a no-op as consuming bytes from the head of the buffer simultaneously makes room at the tail. However, this has some caveats:

  • It is only available on target platforms with virtual memory support, namely fully fledged OSes such as Windows and Unix-derivative platforms like Linux, OS X, BSD variants, etc.

  • The default capacity varies based on platform, and custom capacities are rounded up to a multiple of their minimum size, typically the page size of the platform. Windows’ minimum size is comparably quite large (64 KiB) due to some legacy reasons, so this may be less optimal than the default capacity for a normal buffer (8 KiB) for some use-cases.

  • Due to the nature of the virtual-memory trick, the virtual address space the buffer allocates will be double its capacity. This means that your program will appear to use more memory than it would if it was using a normal buffer of the same capacity. The physical memory usage will be the same in both cases, but if address space is at a premium in your application (32-bit targets) then this may be a concern.

Modules§

  • policy
    Types which can be used to tune the behavior of BufReader and BufWriter.

Macros§

  • do_read
    Shorthand for return DoRead(bool) or return DoRead(true) (empty invocation)
  • flush_amt
    Shorthand for return FlushAmt(n) or return FlushAmt(0) (empty invocation)

Structs§

  • BufReader
    A drop-in replacement for std::io::BufReader with more functionality.
  • BufWriter
    A drop-in replacement for std::io::BufWriter with more functionality.
  • Buffer
    A deque-like datastructure for managing bytes.
  • IntoInnerError
    The error type for BufWriter::into_inner(), contains the BufWriter as well as the error that occurred.
  • LineWriter
    A drop-in replacement for std::io::LineWriter with more functionality.
  • Unbuffer
    A Read adapter for a consumed BufReader which will empty bytes from the buffer before reading from R directly. Frees the buffer when it has been emptied.

Functions§

  • copy_buf
    Copy data between a BufRead and a Write without an intermediate buffer.
  • set_drop_err_handler
    Set a thread-local handler for errors thrown in BufWriter’s Drop impl.